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NP/CLP EQUIVALENCE: A PHENOMENON HIDDEN AMONG 
SPARSITY MODELS FOR INFORMATION PROCESSING 

PENG JIGENt, YUE SHIGANG^, AND LI HAIYANG§ 


Abstract. It is proved in this paper that to every underdetermined linear system 
Ax = b there corresponds a constant p{A, b) > 0 such that every solution to 
the Zp-norm minimization problem also solves the /o-norm minimization problem 
whenever 0 < p < p(A, b). This phenomenon is named NP/CLP equivalence. 


1. Introduction 

In sparse information processing, the following minimization is commonly em¬ 
ployed to model basic sparse problems such as sparse representation and sparse 
recovery, 

(Pq) min||x||o subject to Ax = b (1.1) 

X 

where A is a real matrix of order m x n with m < n, 6 is a nonzero real vector of 
m-dimension, and ||a;||o is the so-called /o-norm of real vector x, which counts the 
number of the non-zero entries in x POSIES]. Unfortunately, although the /o-norm 
provides a very simple and essentially grasped notion of sparsity, the optimization 
problem (Pq) is actually NP-Hard and thus quite intractable in general, due to the 
discrete and discontinuous nature of the Zo-norm. Therefore, many substitution 
models for (Pq) have been proposed through relaxing as the evaluation 

function of desirability of the would-be solution to (Pq) (see, for example, [H [101 HSl 
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m^, and references therein). With the following relationship 

n 

||x||o = lim y^ \xi\P= lim \\x\\l, Wx = {xi,X 2 , ■ ■ ■ ,Xn)^, (1-2) 

p—>-0+ ^ 

2 = 1 

the following minimizations seem to be among the most natnral choices, 

(Fp) min ||a;||^ snbject to Ax = b (1.3) 

where 0 < p < 1. Indeed, the above optimization models, particnlarly for the special 
case when p = 1, have gained its popnlarity in the literatnre (see, e.g., HlElElEni 
[221E11E61E7]), since the remarkable work done by Donoho and Hno [9] and Candes 
and Tao[S] for p = 1 and the initial work by Gribnoval and Nielsen [TS] for 0 < 
p < 1. However, with respect to these choices, a central problem is to what extent 
the minimizations (Fp) can achieve the same results as the initial minimization 
(Fo). A lot of excellent theoretical work (see,e.g.,[ll El dH [18]), together with some 
empirical evidence (see, e.g., i), have shown that the /i-norm minimization (Fi) 
can really make exact recovery provided some conditions such as the restricted 
isometric property (RIP) are assumed. As an original notion RIP has received 
much attention, and has already been tailored to the more general case when 0 < 
p < 1 (see, e.g., inisiiis]). Among those publications mentioned, we would like to 
especially refer to the excellent work done by Donoho and Tanner in [TO], there they 
expose such an amazing phenomenon by means of convex geometry that for any real 
matrix A, whenever the nonnegative solution to (Fq) is sufficiently sparse, it is also 
unique solution to (Fi). That is, there exists a certain equivalence between (Fq) and 
(Fi). As the former is discrete and so NP-Hard and while the latter is continuous 
and equivalent to a linear programming (LP), this phenomenon was called NP/LP 
equivalence therein. It is worthwhile to note that (Fq) and (Fi) are just the extremes 
of (Fp) with respect to p in the interval (0,1), and that the relationship fll.31) together 
with its geometric illustration in Figure 1 appears to indicate the more aggressive 
tendency of (Fp) to drive its solution to be sparse as p decreases. So, it is natural 
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to believe that there exists more general equivalence between (Pq) and (Pp) for 
p G (0,1). Motivated by this, we in this paper aim to expose the equivalence. 



Figure 1. The behaviour of /p-norm for various values of p. See,e.g. m p-38] 

The remainder of this paper is organized as follows. In Section 2 we hrst set up 
a decomposition of the background space M"" with respect to the system Ax = b, and 
then derive constructions and locations of solutions to (Pp) based on the classical 
Bauer maximum principle. Section 3 is devoted to proving the main theorem, which 
establishes an equivalence between (Pp) and (Po), named NP/CLP equivalence. 
Finally we conclude this paper in Section 4. 

For convenience, in this paper we denote by M” the n-dimension real space, 
and for a vector x G M”' by x* its component and by |x| its module vector 
(i.e., |x| = (|xi|, |x 2 |, • • • ,\xn\Y)- We also use M" to represent the positive cone 
{x G M"- : Xj > 0, i = 1, 2, • • • , n}. 

2. Preliminaries: Constructions and Locations of Solutions to (Pp) 

As a preliminary section, this section is devoted to explore how to construct the 
solutions to the problems (Pp) and where the solutions locate. For the minimization 
problems (Pp) are subject to underdetermined linear systems, we hrst are going to 
analyze the constructions of solutions to linear equation Ax = b, where A be an 
m X n real matrix with m < n and b an n-dimension real vector. Without loss of 




4 


generality, we assume throughout this paper that A has full row rank (otherwise, 
we make row transformations simultaneously in both sides of the equation Ax = b, 
resulting in an equivalent equation Aix = b with Ai being of full row rank). So, 
by the nature of geometric aspect of A, we can get a construction decomposition of 
solutions to Ax = b as below. 

Lemma 1. Denote by N{A) the null space of A, and R{A^) the range of A"’" of the 
transposition of A. Then, there holds the following space decomposition 

= N{A)®R{A^), (2.1) 

that is, for every x G M" there uniguely exist x^v G N{A) and xr G R{AA) such that 
X = xn + Xr. Therefore, every solution to Ax = b can be explicitly expressed as 
X = xn + A^{AA^)~^b with xn G N{A). 

Proof. It is clear to see that R{A^) as a linear subspace of M"' is closed since A is 
of full row rank. So, in order to prove the decomposition formula fl2.ip it suffices to 
show that N{A) is just the orthogonal complement of RlA^), i.e., N{A) = R{A'^)-^. 
Obviously, N{A) C R^A^)-^. For the converse containing, let z G R{A'^)-^. By 
dehnition, we have that {Az,y) = {z,A'^y) = 0 for all y G M™', which implies 
that Az = 0, namely 2 ; G N{A). Thus, R{A'^)-^ C N{A) by the arbitrariness of 
^ G R{A^)^. 

Let X be a solution to Ax = b. Then, by fl2.ll) there uniquely correspond 
xat G N{A) and y G such that x = xat + A^y. By dehnitions, b = Ax = AA^y 
and hence x = xat + A^y = xn A AA{AAA)~A, as claimed. □ 

In the above proof, we have used the notation (x, y) to represent the inner 
product of real vectors x and y. For sake of convenience, we will adopt it hereafter. 

Remark 1. The space decomposition implies that each xjq E N{A) can be 

written into the form xn = h — A^{AAA)~^Ah with h = xn + A^b, and so that 
the null of A has such a parameterized expression as N{A) = {h — A^{AA^)~^Ah : 
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h G M""}. ^45 a result, the eonstrained minimizations {Pp) for all p > 0 can be 
equivalently transformed into the global minimizations as follows 

min \\h + A^iAA^y^b-AhW. ( 2 . 2 ) 

On the other hand, it follows by Lemma 1 that for all p > 0 the optimal values to 
{Pp) are upper bounded by \\AA{AAA')~^h\\^, ie.,||x||^ < WAA{AAA)~^h\\^ for every 
solution X to {Pp). This means that the {Pp)s are actually constrained within, ad¬ 
dition to Ax = b, a bounded subset, for example, the l°° ball Boo{r) := {a: e M"" : 
^ rO = 1, 2, • • • , n} with r = n- sup \{A^{AA^)~y)i\. That is, {Pp) with p > 0 

l<i<n 

is equivalent to the following minimization problem 

) min ( 2 . 

Ax=b^x€iBoo (r) 

It is valuable to note that the optimal problem l{2.d\} is subject to a special constraint 
set, which is called typical polytope in term of the following definition. 



Definition 1. [15] A polyhedron G in M"" is a subset as the intersection of finitely 
many halfspacs, where halfspace refers to a set of the form {x G M” : {h, x) < 7 } 
for some vector h E and real number 7 G M. Moreover, a polyhedron is called 
polytope if it is bounded. 


By definition, a polyhedron can be compactly expressed as G = {x G M” : 
Hx < g} for some matrix H and vector of a certain dimension, where Hx < 
g means that the corresponding inequalities hold for all scale components (i.e., 
{Hi,x) < gi, where Hi is the A row of H. Below we always adopt this notation). 
Moreover, it is clear that a polytope is closed and convex, and the intersection of 
any hnitely many polyhedrons is a polytope as long as some of those polyhedrons 
is bounded. Obviously, the /qo ball Boo{r) (see Remark 1), as well as the set of 
solutions to Ax = 6 , is a polyhedron, and so its intersection O = {x G M” : Ax = 
^ ''"0 = 1) 2, • • • , u} is a polytope. Indeed, if let e* denote the A normal base 
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vector of M"" and define a 2 (n + m) x n matrix H as 


H ={ei, -Ci; 62 , - 62 ; ■ ■ ■ ; e„, -e„; ,-A^'f 


(2.4) 


and a 2(n + m)-dimension real vector g as 

9 = (r, r; r, r; ■ ■ ■ ; r, r; V, -V)'^, 


(2.5) 


then the intersection set is compactly expressed by = {x G M"" : Hx < ( 7 }, a 
standard polytope. 

According to convex polytope theory na, we know that polytope possesses snch 
an important characteristic that it can be generated by convexifying a set of hnite 
nnmber of points, which are no other than extreme points. In convexity analysis, 
extreme points always play vital roles. For example, the optimal solntion to a linear 
programming, whose constraint set is generally a polytope (or polyhedron), can be 
achieved at some extreme point (see, e.g., [niEa] and references therein). Below we 
hrst recall the dehnition of extreme point. 

Definition 2. [231 Chapt.8] Let Q be a set of a vector space, and let x* G fi. 
Then, x* is called extreme point of if it does not lie in the interior of any line 
segment entirely contained in (he., x* necessarily coincides with xi orx 2 whenever 
X* = axi + (1 — a;)x 2 with Xi, X 2 G and a G [0,1]). 

The famons Klein-Milman theorem [T] tells that every compact convex snbset 
of a locally convex topological vector space necessarily possesses extreme point (s) 
and it is jnst the closed convex hnll of those extreme points. As a particnlar case, 
Minkowski-Caratheodory theorem [2^ Chap.8, p.l26] states that every point from 
a compact convex snbset of M"’ is a convex combination of at most n + 1 extreme 
points. It is well-known that a polytope possesses at most hnite nnmber of extreme 
points (indeed, if G = {x G M” : Hx < S'} is a polytope, then it has at most 
2 ”^ extreme poins, where m is the dimension of g. See, e.g., iza). We previonsly 
mentioned that a linear programming can attain its optimal valne at some extreme 
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point of the constraint set. In fact, H. Bauer [2] had proved this assertion early in 
1960 for a general convex maximization problem with compact constraint set. We 
state it as a lemma below. 

Lemma 2. (Bauer’s Maximum Principle [I] Chapt.7, p.298]j If G is a compact 
convex subset of locally convex Hausdorff vector space, then every continuous convex 
functional on G has a maximizer that is an extreme point of G. 

Recall that a functional / on a convex set G is said to be concave if there 
holds the inequality af{x) + (1 — a)f{y) < f{ax + (1 — ay)) for all x,y E G and 
a G (0,1). In this terminology, Lemma 2 means that every continuous concave real 
functional dehned on a compact convex subset of locally convex Hausdorff vector 
space achieves its minimum value at some extreme points. By dehnitions, a linear 
function is convex but also can be considered as being concave. This is just why 
a linear programming problem always reaches its optimal value at some extreme 
point, whether it is of minimization problem or maximization one. Consider the 
minimization problem (Pi) under the additional nonnegative constrains a; > 0, or 
equivalently the following linear programming 

(LP) min(l,x) subject to Ax = b,x>0, (2.6) 

X 

where 1 G M"' is the vector whose components are all one. Due to the nonnegativity 
assumption on the variables x, it is easy to show that the constraint set is actually 
bounded and hence a polytope. So by Lemma 2 and Minkowski-Caratheodory the¬ 
orem it is clear that the minimization problem (Pi) is solvable by searching for the 
extreme points of the constraint set. However, the extreme point set possesses up 
to 2™ members, so such searching throughout all extreme points is a overwhelming 
task for large m, which may be an implicit reason why (Pq) is equivalent to (Pi) in 
some cases. 

Before close this section, we present a further remark on Lemma 2, which will 
be useful in proof of our main result. 


Remark 2. Recall that a concave function f is said to be strict if the equality in 
af{x) + (1 — a)f{y) < f{ax+ (1 — ay)) is available only for x = y. In this meaning, 
it is easy to check that every minimizer of a strictly concave function on a compact 
convex set G necessarily exists at a certain extreme point of G. Obviously, the 
function fp{x) = ||a:||^ with 0 < p < 1 is strictly concave in the positive cone M”. 

3. Main Result: Equivalence Between Minimizations (Pq) and (Pp) 

With the preparations given in the last section, we in this section establish an 
eqnivalence between (Pq) and (Pp). To this end, we hrst tnrn to the minimization 
problems (Pp) for 0 < p < 1 and present the following lemma. 

Lemma 3. Let A be an m x n real matrix of full row rank, and b G M”. Given r > 0 
and define a subset of ML as 

G{r) = {z E ML : yi = 1,2, ■ ■ ■ ,n,0 < Zi < r, and 

there is a solution x to Ax = b such that |x| < z}, (3.1) 

where |x| stands for the module vector of x. Then, G{r) is a polytope in M"’. 

Proof. Obvionsly, G{r) is bonnded and closed. Below we prove that G{r) is a poly¬ 
hedron. According to Lemma 1 (combined with Remark 1), we know that x G M" 
solves the system Ax = 6 if and only if it bears the form x = h — {AA'^)~^Ah -|- 

A'^{AA'^)~^b for some h G M”. Denote by A the set of those vectors {z'^,hA)'^ of 
M"" X M"" satisfying the following two ineqnalities 

-z-{I- A^{AA^)-^A)h < A^{AA^)-\ (3.2) 

and 

-z+{I- A^{AA^)-^A)h < -A^{AA^)-A. (3.3) 

Obvionsly, A is a polyhedron of the prodnct space M"" x M"". Let P be the projection 
from M"' X M"' to the hrst part, i.e., P{z, y) = z. Then, by the projection property of 
polyhedron [13] we know that P(A) is also a polyhedron of M^. Therefore, to close 
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the proof, we only need to show that G{r) = P(A) n (r), where B'^{r) stands 
for the subset of nonnegative elements of Boo{r). 

In light of the above discussion, it is clear that G{r-) C P(A) fl B^{r). For the 
converse containing relation, let z G P(A) fl B'^{r). Then, 0 < < r for all i = 

1, 2, • • • , n, and there corresponds a h G M” such that G A, that is, the pair 

{z, h) satishes both the inequalities fl3.2p and fl3.3p . Let x = h + A'^{AA'^)~^{b — Ah). 
Then, it is a routine work to check that x solves Ax = b, and moreover it follows from 
the inequalities fl3.2p and fl3.3p that |x| < 2 :. So z G G{r). The proof is therefore 
completed. □ 

Figure 2 displays basic shapes of G{r) in the plane M^, except several degenerate 
cases (e.g., the segment between these two points (0,r)^ and (r,r)^). 



Figure 2. Shapes of the polytope G{r) dehned as in fl3.ip in the plane. 

In order to prove the following theorem, the main result of this paper, we make 
a remark on the solutions to (Po)- Due to the integer-value virture of /o-norm, the 
optimal value of (Pq) is certainly achieved in a bounded set. That is, there exists a 
constant ro > 0 such that 

min ||x||o = min ll^^llo- (3-4) 

Ax=b Ax=b,xGBoo{ro) 

(Geometrically, it appears to be true that the optimal solutions to (Pq) locate at 
those intersections of the plane Ax = b with coordinate axises or coordinate planes). 

Theorem 1. There exists a constant p{A,b) > 0 such that, whenever 0 < p < 
p{A,b), every solution to (Pp) also solves (Pq). 
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Proof. Let G{ri) be defined as in fl3.ip with ri = n- sup \{A{AA'^) ^b)i\. Then, by 

l<i<n 

Lemma 3 we know that G(ri) is a polytope and hence has finite number of extreme 
points. Denote by E{G{ri)) the set of extreme points of G'(ri), and define a constant 
rm(A,b) as follows 


rm{A,b) = min 2 ;*. (3.5) 

zGE(G(ri)),Zi^0,l<i<n 

Clearly, the defined constant r^iA, b) is finite and positive (i.e., 0 < rm(^, b) < 00 ), 
due to the finiteness of E{G{ri)). (Here we draw attention to that rm{A,b) only 
depends on A and b. However, for simplicity we may draw off A, b from r^iA, b) in 
the following discussion). 

Let tq be given as in fl3.4|) . and r = max{ro,ri}. Then, we have G{ri) C G{r), 
and similar to fl2.3p we have 


min ||x||!( = min |la:|l?’ = min ||a;||ll. (3.6) 

Ax=b Ax=b,x£Boo{ri) Ax=b,x^BoG(r) 


and 


min ||x||o = min ll^^llo = Iloilo- (3-7) 

Ax=b Ax=b,xGBoo{PQ) Ax=b,xGBoc,(p) 

Any given a solution Xp to (Pp). By Remark 1 we have Xp G Now, 

let Zp = \xp\. Then, by Lemma 3 it is clear to see that Zp G G'(ri) and solves the 
following minimization problem 



n 


min ID 

z£G{ri) 



(3.8) 


Moreover, since f{z) = ||z||^ is strictly concave in G{ri), it follows by Lemma 2 and 
its Remark 2 that Zp G E{G{ri)). Hence, by noticing the fact that the mapping 
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p ^ is decreasing for x G (0,1) and while increasing for a: > 1, we have 


la^pllo = \\zp\\o = Yim[zl^ + zl2 + --- + zlJ\ 

Q-\X) 


= lim 

gl-O 


< 


Zp\ 

I’m 

P 


+ 

Tm/ \ r, 


+ 

Zp2 


Zp2 

Tra 

P 


+ ••■ + 


+ ■ ■ • + 


'^pn 


Zpn 

Tm 

P 




— p P rnin 2^ ^ 

' m .ll^llp m . 

zeG(ri) Ax=b,xSBoo(ri) 


mm 


Uc 


_ p-p 
' m 


r 


mm \\x\\^ = ( — 1 min 
Ax=b,xGBao{r) V I’m/ Ax=b,x£Bco(r) 


< 


^ y ■ II -1 II 

— mm ||r x||o 

"^m/ Ax=b,x&Boo(r) 
r 

— mm Iloilo = 

r^J Ax=b,x&Boo(r) \r„ 


mm ||x||o, 
Ax=b 


where the last equality follows from fl3.7p . Because ||xp||o is an integer number, from 
the above inequality it follows that ||xp||o = rnin ||a:||o (that is, Xp solves (-Pq)) when 

Ax=b 


(3.9) 


— min \\x\\o < min \\x\\o + 1 
^ J Ax=b Ax=b 

Obviously, the above inequality is true whenever 

In ( min ||a;||o + 1 ) — In ( min ||a;||o 
V Ax=b / V Ax=b / 

Therefore, denoting by p{A, b) the right side of the above inequality, we conclude 
that when 0 < p < p{A,b), every solution Xp to (Pp) also solves (Pq). The proof is 
thus completed. □ 


Remark 3. From the proof it is clear to see that the parameters r^ and ri are just 
used to bound the ranges of solutions to (Pq) and (Pi) respectively. So, to obtain 
a better p{A, b) we can replace the number r in the inequality H3.10\) with another 
number that bounds the solutions to (Pp) for all 0 < p < 1. 

As is well known, (Pq) is combinatorial and NP-hard in general, and while 
(Pp) for p > 0 is continuous and perhaps is polynomially computable. In this 
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sense, the significance of the theorem lies in that it really bridges the gap between 
a combinatorial problem and a continnons one. To highlight the NP natnre of (Pq) 
and the continnity featnre of (Pp), we name the phenomenon stated by Theorem 1 
‘^NP/CLP eqnivalence”. Correspondingly, we call the maximal p{A,b) ^^NP/CLP 
eqnivalence constant”, and denote it by p*{A,b). 

Obvionsly, it is important to evalnate the NP/CLP eqnivalence constant p*{A, b) 
for ns to choose an appropriate model (Pp) snbstitnting for (Pq). However, this is 
a hard and difficnlt work, althongh the ineqnality fld.lOp can be nsed to derive a 
rndimentary estimation. With the relationship fll.2p . perhaps someone believes that 
the constant p{A, b) is determined by some single valne (that is, if (PpJ is eqnivalent 
to (Pq), so are all (Pp) for p < pi). Nevertheless, the following example shows that 
it is not trne. 


Example 1. Consider the minimization (Pp) with respect to the underdetermined 
linear system Ax = b with 



A = 


It is easy to show that the solutions x = 
terized form 


0 \ 

1 , 6= (1,2,3)^. (3.11) 

(ail, X 2 , X 3 , Xi)^ have the following parame- 


Xi = t,X2 


4 40 





(3.12) 


where the parameter t varies in R. Hence, it can he the Ip-norm is computed by the 
following function of t, 


HS = l<l’’ + 


4 40 

p 

29, 20 , 

p 

58 , 20 , 

-^- 1 

27 27 

+ 

— 1- 1 ) 

9 ^ 29 ^ 

+ 

(1 t) 

135^ 29 ^ 


(3.13) 


Obviously, the unigue solution to (Pq) is xq = (1.45,2,0,0)^ (with respect to t = 
1.45j, and all the solutions to {Pp) for 0 < p < 1 exist in the set Boo{2). It is 
also easy to test that the constant rm{A,h) defined as in LS. .5)) eguals to 0.1. Hence, 
by the ineguality H3.10\) we can derive that p*{A,b) > 0.135. So, by Theorem 1 we 
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know that Xq = (1.45,2,0,0)^ is the unique solution to {Pp) for 0 < p < 0.135. 
From Figure 3 it is seen that the lom-iT-orm and lo,i 35 -norm reach their minimums 
at t = 1.45, which corresponds to the unique solution Xq = (1.45, 2, 0, 0)^. 

Now we consider three cases when p = 0.8, 0.95, 1, respectively. The behaviours 
of ||x||() as the functions oft are displayed in Figure 3 for those p. By the formula 
iS.13\) it is easy to test that xq.s = (0 .1, 0, 3, 0.4)"^ solve {Pq.s), and while xo.gs = 
Xi = (1.45,2,0,0)^ solves both (P0.95) (-^1)? respectively. But Hxq.sIIo = 3 > 

ll^^o.gsll = 2 = ||xo||o- This shows that p*{A, b) < 0.8 in spite of the fact that (P0.95) 
possesses the same unique solution as {Pq)- 



Figure 3. Behaviours of the /p-norms as the functions of t for p = 
0.1,0.135,0.8,0.95 and 1, respectively. 

Obviously, the above example also incidentally shows that in the whole interval 
(0,1) of p it is not true that the smaller p is, the sparser the solutions to (Pp) are. 

4. Conclusions 

Among the numerous substitution models for the Iq minimization problem (Pq), 
the /p-norm minimizations (Pp) with 0 < p < 1 have long been considered as the most 
natural choices. However, it has not been answered to what extent these models 
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(Pp) can replace (Pq). In this paper, we have exposed the equivalence between (Pq) 
and (Pp), so completely answered the question mentioned. The established equiv¬ 
alence means that solving (Pq) can be completely attacked by solving the convex 
minimization (Pp) instead for some small p, while the latter is computable by some 
commonly used means at least for some special p. However, it should be pointed 
out that the main result obtained in this paper is only qualitative, it has not given 
quantitative characterization to the NP/CLP equivalence constant, which the au¬ 
thors think is important for model choice and then for algorithm design. Altogether, 
the authors wish this paper can throw away a brick in order to get ages. 
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